Managing a data center

ABSTRACT

A data center system includes a frame assembly that includes a plurality of bays defined along a lengthwise dimension of the frame assembly, the plurality of bays arranged in a plurality of stacked layers of bays, the plurality of stacked layers including at least a first layer of bays and a second layer of bays positioned vertically above the first layer of bays; a plurality of server racks positioned in the bays of the first layer of bays, each of the server racks configured to support a plurality of data center server devices that define a particular amount of computing power; and a plurality of network switches positioned in the bays of the second layer of bays, each of the network switches communicably coupled to at least one of the data center server devices in the first layer of bays.

TECHNICAL FIELD

The present disclosure relates to systems and methods for managing adata center and, more particularly, managing networking connections andservicing of electronic computing devices in a data center.

BACKGROUND

Planning for and provisioning physical network infrastructure resourcesfor a large scale data center deployment is a challenging task.Co-locating networking switches and information technology (IT)equipment racks in the same space results in inefficiencies and reducedflexibility. For example, networking provisioning may causeunderutilization of power, cooling, and space resources. Further,networking equipment often has higher availability requirements due totheir potential to impact larger failure domains. In some cases, complexnetworking cable routes and non-optimal cable path lengths occur due tocable tray conveyance in multiple elevations in a grid configurationabove IT equipment. Such circumstances may also cause inflexibility toexperiment with alternative network topologies.

SUMMARY

In a general implementation, a data center system includes a frameassembly that includes a plurality of bays defined along a lengthwisedimension of the frame assembly, the plurality of bays arranged in aplurality of stacked layers of bays, the plurality of stacked layersincluding at least a first layer of bays and a second layer of bayspositioned vertically above the first layer of bays; a plurality ofserver racks positioned in the bays of the first layer of bays, each ofthe server racks configured to support a plurality of data center serverdevices that define a particular amount of computing power; and aplurality of network switches positioned in the bays of the second layerof bays, each of the network switches communicably coupled to at leastone of the data center server devices in the first layer of bays.

In an aspect combinable with the general implementation, the pluralityof bays are further arranged in a third layer of bays that is positionedvertically between the first and the second layers of bays.

In another aspect combinable with any of the previous aspects, anotherplurality of server racks are positioned in the bays of the third layerof bays, each of the other plurality server racks is configured tosupport another plurality of data center server devices that define anadditional amount of computing power.

Another aspect combinable with any of the previous aspects furtherincludes at least one cooling unit sized for a bay of the plurality ofbays of the frame assembly and configured to circulate a heated airflowfrom an open back side of a particular server rack, cool the heated air,and circulate a cooling airflow through an open front side of theparticular server rack.

In another aspect combinable with any of the previous aspects, each bayincludes a volume defined at least in part by a specified height that isorthogonal to the lengthwise dimension and a specified width that isparallel to the lengthwise dimension and sized to at least partiallyenclose at least one server rack.

In another aspect combinable with any of the previous aspects, thespecified width and the specified height of the bay are based on a formfactor of the at least one server rack.

In another aspect combinable with any of the previous aspects, at leastone perimeter dimension of the cooling unit is based on a form factor ofthe at least one server rack.

In another aspect combinable with any of the previous aspects, the atleast one server rack includes two server racks, each of the two serverracks providing between 5 kW and 100 kW of computing power.

In another aspect combinable with any of the previous aspects, at leasta portion of the frame assembly that defines the bays of the secondlayer of bays is configured to mount on a support surface of a datacenter building.

In another aspect combinable with any of the previous aspects, thesupport surface includes a structural steel mezzanine.

In another aspect combinable with any of the previous aspects, the frameassembly includes a first portion that defines the first layer of bays;a second portion stacked on top of the first portion that defines thesecond layer of bays; and a third portion stacked on top of the secondportion that defines the third layer of bays.

In another general implementation, a method of managing a data centerincludes operating a plurality of data center server devices that definea particular amount of computing power, the plurality of data centerserver devices supported in a plurality of server racks positioned in aframe assembly that includes a plurality of bays defined along alengthwise dimension of the frame assembly, the plurality of baysarranged in a plurality of stacked layers of bays, the plurality ofstacked layers including at least a first layer of bays and a secondlayer of bays positioned vertically above the first layer of bays, theplurality of server racks positioned in the bays of the first layer ofbays; and operating a plurality of network switches positioned in thebays of the second layer of bays, each of the network switchescommunicably coupled to at least one of the data center server devicesin the first layer of bays.

In an aspect combinable with the general implementation, the pluralityof bays are further arranged in a third layer of bays that is positionedvertically between the first and the second layers of bays.

Another aspect combinable with any of the previous aspects furtherincludes operating an additional plurality of data center server devicesthat define an additional amount of computing power, the additionalplurality of data center server devices mounted in an additionalplurality of server racks positioned in the bays of the third layer ofbays.

Another aspect combinable with any of the previous aspects furtherincludes operating at least one cooling unit sized for a bay of theplurality of bays of the frame assembly to circulate a heated airflowfrom an open back side of a particular server rack, cool the heated air,and circulate a cooling airflow through an open front side of theparticular server rack.

In another aspect combinable with any of the previous aspects, each bayincludes a volume defined at least in part by a specified height that isorthogonal to the lengthwise dimension and a specified width that isparallel to the lengthwise dimension and sized to at least partiallyenclose at least one server rack, and the specified width and thespecified height of the bay are based on a form factor of at least oneserver rack.

In another aspect combinable with any of the previous aspects, at leastone of the plurality of server racks includes two server racks, each ofthe two server racks providing between 5 kW and 100 kW of computingpower.

Another aspect combinable with any of the previous aspects furtherincludes deploying the frame assembly into a human-occupiable space of adata center building.

Another aspect combinable with any of the previous aspects furtherincludes arranging a first portion of the frame assembly that definesthe first layer of bays onto a data center floor of the data centerbuilding.

Another aspect combinable with any of the previous aspects furtherincludes arranging a third portion of the frame assembly that definesthe third layer of bays on top of the first portion of the frameassembly.

Another aspect combinable with any of the previous aspects furtherincludes arranging a second portion of the frame assembly that definesthe second layer of bays on top of the third portion of the frameassembly.

Another aspect combinable with any of the previous aspects furtherincludes deploying at least one least one cooling unit into at least onebay of the plurality of bays of the frame assembly.

Another aspect combinable with any of the previous aspects furtherincludes deploying at least the plurality and the additional pluralityof server racks into the first and third layers of bays defined by thefirst and third portions of the frame assembly.

Another aspect combinable with any of the previous aspects furtherincludes deploying the plurality of network switches into the secondlayer of bays defined by the second portion of the frame assembly.

Another aspect combinable with any of the previous aspects furtherincludes communicably coupling the plurality of network switches withthe plurality and the additional plurality of server racks.

Another aspect combinable with any of the previous aspects furtherincludes mounting at least a part of the second portion of the frameassembly to a support surface of the data center building.

In another aspect combinable with any of the previous aspects, thesupport surface includes a structural steel mezzanine.

Another aspect combinable with any of the previous aspects furtherincludes deploying, to at least one of the first or third layers of baysin the first and third portions of the frame assembly, an incrementalamount of data center server devices supported in an incremental numberof server racks.

Another aspect combinable with any of the previous aspects furtherincludes based on the deploying of the incremental amount of data centerserver devices, deploying an incremental amount of network switches tothe second layer of bays in the second portion of the frame assembly.

Another aspect combinable with any of the previous aspects furtherincludes communicably coupling the incremental amount of networkswitches to the incremental amount of data center server devices.

Another aspect combinable with any of the previous aspects furtherincludes based on the deploying of the incremental amount of data centerserver devices, deploying at least one additional cooling unit into atleast one additional bay of the plurality of bays of the frame assembly.

Implementations according to the present disclosure may include one ormore of the following features. For example, implementations accordingto the present disclosure may provide for a dedicated networking spacethat allows a more tightly packed aggregation of networking switches anda reduction total cable lengths. As another example, implementationsaccording to the present disclosure may increase a speed and efficiencyof deploying networks within a data center among and between informationtechnology (IT) equipment (e.g., electronic devices such as servers thatare primarily tasked with consumer related jobs such as Internetsearches, electronic mail, cloud computing, and otherwise). In someimplementations, network deployments can be sped up due to simpler andshorter cable runs between network switches and IT equipment. As anotherexample, implementations according to the present disclosure may allowfor a reduced latency of communication among IT equipment and, e.g.,networks external to the data center, by connecting the IT equipmentwith network resources through shorter, more direct cabling runs ascompared to conventional data centers. This may be especially true forhigh performance IT equipment. As another example, implementationsaccording to the present disclosure may more efficiently evaluatevarious networking topologies without disturbing IT equipmentdeployments before such topologies are actually implemented. Further,power and cooling solutions for an IT equipment layer (or layers) and anetwork switch layer can be sized and implemented according to theiravailability targets, which are usually different. Moreover, resourcescan be fully dedicated to IT equipment instead of being de-rated dueadditional networking requirements. In some implementations, totalnetworking cable path lengths can be reduced by placing aggregationnetworking switches closer together. In the case of fiber optic cabling,a reduction in cable length may enable the use of more economicallyefficient optical fiber cables and optical fiber transceivers. Further,more economically efficient transceivers, active optical cables, or bothmay be used due to implementations according to the present disclosure.

Implementations according to the present disclosure may include one ormore of the following features. For example, installation of networkingcabling between networking switches and IT equipment can be shortenedand simplified and may lend itself to motorized, pulley-driven cableruns as opposed to manually installed cable runs to reduce networkingdeployment time. Further, such implementations may also enable the useof pre-fabricated drop-in networking cable assemblies to reducenetworking deployment time. Further, implementations according to thepresent disclosure may provide for an independently reconfigurablenetworking layer without disturbing the IT layers. In addition,implementations according to the present disclosure may provide for thepotential for variable power and thermal domains within the data center.

The details of one or more implementations of the subject matterdescribed in this disclosure are set forth in the accompanying drawingsand the description below. Other features, aspects, and advantages ofthe subject matter will become apparent from the description, thedrawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an isometric illustration of a data center that includesstacked rows of IT equipment according to the present disclosure.

FIGS. 2A-2B are schematic illustrations of a side and end view,respectively, of an example implementation of a data center system thatincludes an IT equipment layer and a networking switch layer accordingto the present disclosure.

FIGS. 3A-3B are schematic illustrations of a side and end view,respectively, of another example implementation of a data center systemthat includes multiple IT equipment layers and a networking switch layeraccording to the present disclosure.

FIGS. 4A-4B are schematic illustrations of a side and back view,respectively, of an example implementation of a data center system thatincludes an automatic servicing machine according to the presentdisclosure.

FIG. 5 is a schematic illustration of an example controller for a datacenter IT equipment servicing system according to the presentdisclosure.

DETAILED DESCRIPTION

A data center according to the present disclosure includes multiple(tens, hundreds, thousands, tens of thousands) electronic components,such as servers, processors, memory modules, network switches, and othercomponents. The components, in some aspects, are arranged in serverracks or network racks that support (e.g., physically, with electricity,with cooling) the components. For example, server racks and networkracks may be arranged in rows that include two or more stacked layers ofthe components. In some aspects, certain components, such as informationtechnology (IT) components (e.g., servers, data storage) are arranged inrows of one or more layers with communicably coupled networkingcomponents (e.g., switches) arranged in a layer separate from the ITcomponent layers (e.g., vertically above).

In some aspects, the IT components may be serviced (e.g., deployed intoracks, replaced, repaired, or otherwise) by one or more automatedequipment. In some aspects, the automated equipment may behuman-operated or human-controlled (e.g., moved around the data centerto service the components). In some aspects, the automated equipment maybe computer-controlled (e.g., by specified software instructions) tomove around the data center to service the components. In some aspects,the automated equipment may be computer-controlled according toartificial intelligence (AI) principles in order to move around the datacenter to service the components.

FIG. 1 is an isometric illustration of a data center 100 that includesstacked rows of servers and networking switches. Data center 100includes a data center shell building 102 that defines ahuman-occupiable workspace into which one or more (four shown here)clusters 104 of rows 105 of data center racks 103. As shown, certainclusters 104 of the rows 105 are separated by fire-rated walls 110. Inthis example, a particular cluster 104 includes multiple (e or more)rows 105 of data center racks 103. As shown in the implementation ofdata center 100, each cluster 104 includes eight rows of racks 103. Theeight rows 105 of racks 103 in each cluster 104 includes four pairs ofrows 105 with each pair of rows 105 being arranged as two back-to-backrows 105 of data center racks 103. As explained more fully in thepresent disclosure, two back-to-back rows 105 of data center racks 103may be arranged so that a warm air aisle is defined there between andpositioned to receive an airflow from back sides of the two rows 105 ofdata center racks 103 that has been warmed by many (e.g., tens,hundreds, thousands, tens of thousands within the data center 100) ofheat-generating IT computing devices supported in each rack 103.Further, pairs of rows 105 of data center racks 103 may be separated bya cold air aisle through which a cooling airflow (to cool theheat-generating IT computing devices) is circulated. Thus, in exampleimplementations of the data center 100, back-to-back rows 105 of racks103 are separated by warm air aisles (adjacent open back sides of thedata center racks 103) while pairs of rows 105 of data center racks 103are separated by cold air aisles (adjacent open front sides of the datacenter racks 103).

As further illustrated in FIG. 1, each row 105 of data center racks 103are arranged in multiple (more than one) vertical layers 107. Each layer107 of data center racks 103 includes a horizontal row 105 of the racks103 as determined by a lengthwise dimension (shown here as L) of thedata center shell building 102. Further each layer 107 includes at leastone other layer 106 of data center racks 103 positioned below or above(or both). Thus, layers 107 of the rows 105 of data center racks 103 aredefined and positioned in a vertical direction (shown as H) in the shellbuilding 102; rows 105 of the data center racks 103 are defined andposition in the lengthwise dimension, L; and clusters 104 of the rows105 of the data center racks 103 are defined and positioned along awidthwise dimension (shown as W) in the shell building 102.

In the illustrated implementation of data center 100, each of the layers107 may include data center racks 103 that support IT equipment ornetwork switch equipment that communicably couples (e.g., with fiberoptic or other network cabling) the IT equipment (e.g., to other ITequipment, a network internal to the data center 100, or a networkexternal to the data center 100). For example, as shown in FIG. 1, ITequipment layers 106 are arranged in three of the layers 107, includinga bottom layer 107 (e.g., supported on a floor of the data center 100),a middle layer 107 (e.g., vertically mounted on the bottom layer 107),and a top layer 107 (e.g., vertically mounted on the middle layer 107).Mounted above the top layer 107 of IT equipment layers 106 is anetworking layer 108 that supports network switches (e.g., stage 2switches). In some aspects, the networking layer 108 is positioned on amezzanine of the data center 100 (e.g., a human-occupiable structure ata height of the networking layer 108)

Thus, the data center 100 is an example implementation of a data centerthat includes a physically separate floor for the network aggregationswitches, or stage 2 switches. The networking plane, in data center 100,is located on a separate floor (e.g., mezzanine) above (or a subfloorbelow) a data center floor that supports IT equipment (e.g., servers andstorage machines). In some aspects, moving the networking switches to aseparate area (e.g., layer 108) may eliminate many of the cable traysrequired in the IT equipment area (e.g., layers 106) since most of thecable trays are used to support the network links for network switches(e.g., stage 2) and above.

Data center 100 also includes (not specifically shown in FIG. 1)physical cabling that is routed between the layers 107 (e.g., among ITlayers 106, between IT layers 106 and network layer 108). For example,pre-terminated fiber optic cable bundles can be routed between and amonglayers 107 to directly connect, e.g., a stage 1 switch to a stage 2switch. Alternatively, patch panels can be used as termination pointsbetween IT layers 106 (e.g., that include server and storage machinesand stage 1 switches) and the network layer 108 (e.g., that includesstage 23 switches) with structured fiber optic cabling arranged betweenthe layers 106 and 108.

Data center 100 may also allow for or facilitate an optimized layout ofIT equipment layers 106 and networking layer 108 even though theirrespective requirements are different. The networking layer 108 mayenable a variety of network switch layouts that would be optimized froma network cabling perspective for different topologies (for example,bus, ring, star, tree, mesh, or hybrids). The networking layer 108 thatincludes support structure (e.g., racks 103) for the network switches(e.g., stage 2) can be arranged to reduce a distance between switchingnodes of the network layer 108. For example, cable trays (in the networklayer 108 or between the network layer 108 and IT layers 106) can bedeployed on a tighter rectangular grid or in radial patterns to reducenetwork cable path lengths and simplify network cable routes. On the ITequipment layers 106, data center racks 103 can be arranged in rows tomaximize deployment density. Further, in the networking layer 108 (e.g.,a raised mezzanine or lowered sub-floor), non-traditional cable supportstructures or cabling planes can be used to convey networking cables onthe networking layer 108 with multi-directional runs (e.g., as opposedto conventional cabling runs that may only allow bi-directional layoutsof networking cables). In some aspects, pre-built, large network cablingassemblies can be dropped in and connected to data center racks 103 inthe networking layer 103 that are deployed in closer proximity to eachother in set patterns for faster deployment times.

By taking advantage of a vertical dimension in three-dimensional space(the interior environment of the data center 100) to arrange networkswitches (e.g., stage 2) and IT equipment that need to be closelyinterconnected, improvement in fiber optic cabling inventory for thedata center 100 can be simplified and reachable positions of fiber opticcable length limitations can be maximized. For example, fiber opticcable reach (e.g., how far an operational cable may extend between twopoints) may become shorter as bandwidth increases, and therefore,increasing a number of reachable IT equipment rack spots may increasecable efficiency. By taking advantage of the vertical dimension, theremay be increased capability to land IT equipment or groups of ITequipment in any location in a cluster without having to considernetwork cabling length constraints.

Additionally, a dedicated network layer 108 allows for many operationalbenefits throughout the life cycle of the data center 100. For instance,an increase in network deployment velocity can be expected viaseparation of network cabling routing paths leading to easierincremental network turn-ups, decommissions, and transformations.Further, operations can be accelerated by separation of trades andde-conflicting of areas turned-up by different disciplines. (e.g.network installers, electrical, IT equipment installers).

In particular implementations, a cooling unit that operates within thelayer(s) of mixed computing/cooling equipment may use a cooling liquid(e.g., chilled water or chilled glycol from one or more chillers,condenser water or other evaporatively cooled liquid, or otherwise) froma data center building central plant. The cooling unit utilizes thecooling liquid in an air-to-liquid heat exchanger (e.g., liquid orrefrigerant cooling coil) to cool a flow of heated air from multipleserver racks. The cooling unit may include one or more fans that arecontrollably operated to circulate the heated airflow from the serverracks to the cooling unit, through the air-to-liquid heat exchanger tocool the heated airflow, and from the cooling unit (as a coolingairflow) back, to the server racks.

The layer(s) of mixed computing/cooling equipment and IT equipmentlayer(s) may be deployed in a data center building in an incrementalprocess. For example, an initial amount of IT power load (that generatesa particular heat load) and networking equipment (that may also generatea heat load) may be determined. For an initial deployment, the IT powerload may be in a single zone (e.g., row, cluster, columns, rows andcolumns, or combinations thereof) or in or in multiple zones (e.g.,multiple rows, multiple clusters, multiple layers of rows or clusters).Likewise, the determined networking load may be in a single layer (e.g.,in or on a mezzanine or sub-floor layer) or multiple layers. Based onthe determined IT power load and networking equipment (and correspondingheat load), cooling capacity (e.g., per zone) may be determined, therebydetermining a number of cooling units needed per zone (e.g., based on amaximum cooling capacity per cooling unit). IT equipment (e.g., serverracks) and networking equipment may be deployed in the data center(e.g., within server rack frame assemblies arranged in one or morevertical layers) along with the determined number of cooling units (alsowithin the server rack frame assemblies arranged in one or more verticallayers). For example, the cooling modules may be deployed interspersedin between server racks within the frame assembly, e.g., in particularlocations (e.g., bays) to account for cooling requirements dictated bythe server rack placements.

The initial deployment process described above can be iterative andrepeated for technology refreshes (e.g., replacement of particularserver racks with other, higher/lower power IT server racks ornetworking equipment) or new deployments (e.g., addition of server racksand networking equipment). In each iteration, based on the iterativelydetermined IT load and networking equipment (and correspondingdetermined heat load), a number and/or location of cooling units can bere-calculated. For example, if low powered server racks are beingreplaced with higher (relatively) powered server racks to increase aserver rack power density of the data center or zone of the data center,the cooling capacity may be recalculated and cooling units can be addedfor the additional cooling power. Similarly, cooling units could beremoved when server rack power density is reduced in the data center orzone of the data center. With incrementally deployed cooling units,cooling could be “as needed” and properly sized based on the deployed ITequipment in each zone (e.g., single layer row, portion of a singlelayer row, multi-layer row, or portion of a multi-layer row). Forexample, for low power density server racks, fewer cooling units may bedeployed within a particular zone, while a larger number of coolingunits may be needed for high power density server racks within aparticular zone.

Incremental, mixed computing/cooling equipment deployment can beimplemented in different data center equipment architectures. Forexample, although the present disclosure describes single andmulti-level row-based architectures, other, non-linear architectures(e.g., single or multi-level clusters) can also be implemented accordingto the present disclosure. Further, although layers of mixedcomputing/cooling equipment architectures are shown, the coolingequipment may be extracted from the layers of computing equipment andimplemented, e.g., as overhead cooling, underfloor cooling, end of rowcooling, conventional CRAC cooling, or otherwise.

FIGS. 2A-2B are schematic illustrations of a side and end view,respectively, of an example implementation of a data center system 200that includes a single, mixed computing and cooling layer 211 positionedin a human-occupiable workspace 204 of a data center building 202 with anetworking layer 215 positioned above the computing/cooling layer 211.As shown in this example implementation, the data center system 200includes a layer 211 of server racks 214 and cooling units 216 that arepositioned in a frame assembly 210. In this example, the layer 211 isarranged in a row 208 (e.g., linear or substantially linear arrangementof the frame assembly 210). In alternative arrangements, the layers 211and 215 may be arranged in a non-linear frame assembly, such as, forinstance, a circular or partially circular frame assembly or otherwise.

As shown in FIGS. 2A-2B, the networking layer 215 is positioned on topof the computing layer 211. In this example, the networking layer 215include network switches 230 (e.g., stage 2 switches) positioned withinbays 212 of the frame assembly 210 and connected, via network cabling232 (e.g., fiber optic or otherwise) to the IT equipment in layer 211(e.g., one or more devices in the server racks 214). For example, insome aspects, particular network switches 230 in a particular bay 212 oflayer 215 may be coupled via the network cabling 232 to server racks 214that are located in a particular bay 212 of layer 211 that is directlybelow (e.g., vertically) the particular bay 212 of layer 215. In someaspects, particular network switches 230 in a particular bay 212 oflayer 215 may be coupled via the network cabling 232 to server racks 214that are located in a particular bay 212 of layer 211 that is indirectlybelow (e.g., vertically below and horizontally offset by one bay 212)the particular bay 212 of layer 215. Even further, in some aspects,particular network switches 230 in a particular bay 212 of layer 215 maybe coupled via the network cabling 232 to server racks 214 that arelocated in a particular bay 212 of layer 211 that is indirectly below(e.g., vertically below and horizontally offset by more than one bay212) the particular bay 212 of layer 215. In short, networking switches230 may be connected via the cabling 232 to server racks 214 that arevertically below, as well as vertically below and offset.

In some aspects, networking switches 230 may be relatively largeswitches (e.g., several to tens of RU's (Rack Units) tall or many 1-2RUtall switches). For example, in some aspects, the switches 230 may benetwork fabric racks that are made up of several line cards (with theswitch chips) that plug into a large multi-RU chassis that contains abackplane that connects the line cards in a desired fashion. In othercases, such network fabric racks can be defined using many (12+) 1RUswitches connected to each other in a desired fashion. Racks thatsupport the switches 230 may be closely coupled to build even largerconfigurations of switch networks. Such switch racks may also containpower conversion, batteries, or passive fiber patch panels.

As shown in FIGS. 2A-2B, the networking cabling 232, and thus thecommunication connections between server racks 214 (e.g., through thenetwork switches 230), between server racks 214 and a network externalto the data center 200 (e.g., through network switches 230), and betweennetwork switches 230 (e.g., through server racks 214 or from/to anexternal network), are largely unimpeded by, e.g., coolinginfrastructure, such as cooling units 216, ductwork, piping, electricalcabling, or other cooling equipment, due to the positioning of thenetwork switches 230 in the layer 215. By mounting the networking layer215 vertically above the IT equipment layer 211, the Z, or vertical,dimension of the human-occupiable workspace 204 may be taken advantageof as opposed to conventional data centers. This may also provide fordecreased lengths of the network cabling 232 (e.g., fiber optic orotherwise) between switches 230 and server racks 114, switches 230 and aconnection at the data center 200 to an external network, and betweenserver racks 114.

In some aspects, the example implementation of the data center 200 withseparated (e.g., vertically) IT equipment layer 211 from networkinglayer 215 may also increase a deployment velocity of the data centerequipment (e.g., server racks 214, networking switches 230) byseparating networking routing paths (e.g., for the cabling 232) toenable easier incremental network turn ups, decommissions, andtransformations. This may also improve accessibility for the networkcabling 232. Such decoupling of the IT layer 211 and network layer 215may also allow separate construction trades (e.g., electrical andcabling installation) to occur in parallel.

The frame assembly 210 comprises structural members (e.g., metal ornon-metal, such as composite) arranged and connected to form multiplebays 212 in the frame assembly 210. Each bay 212, in this exampleimplementation, defines a volume within the frame assembly 210 and maybe the same or similar in volume and perimeter dimensions (e.g., height“H”, width “W”, and depth “D”) as the other bays 212 in the layer 211 ofthe server rack assembly 210. In some aspects, H may be about 10 feet, Wmay be about 10 feet, and D may be about 3 feet, as an exemplary set ofdimensions.

In some aspects, the perimeter dimensions of the bays 212 (and thus thevolumes defined by the bays 212) in the frame assembly 210 are based onor associated with dimensions of the server racks 214. For example, inthe illustrated implementations, each bay 212 is sized (e.g., at least Wand H) according to dimensions of a double server rack 214 (i.e., twoserver racks of 42U in height, two server racks of between 18-22U inheight, or other sized server racks). In some aspects, the server racks214 may be 54U in height, although frame assembly 210 may handle avariation of rack sizes. For example, the racks 214 may be about 50inches wide each.

Furthermore, different combinations of racks 214 can be used in theexample implementation. For an example, four racks 214, each 24 inchesin width, can be used within the frame assembly 210. The racks 214 canalso vary in depth. For example, the front faces of the server racks 214may be flush with a front side 222 of the assembly 210, while a rear ofthe racks 214 may extend various depths beyond back sides 224 of theassembly 210.

As shown in FIG. 2A, the server racks 214 support electronic devices,such as processors, memory modules, networking switches, batterymodules, and other server rack computing components, both physically byproviding structure for the devices to be placed in and electrically byproviding electric power to the devices from a main source of power(e.g., through an inverter, a transformer, or both). Generally, eachillustrated server rack 214 may be one of a number of server rackswithin the data center building 202, which may include a server farm ora co-location facility that contains various rack mounted computersystems. Each server rack 214 may define multiple slots that arearranged in an orderly and repeating fashion within the server rack 214,and each slot is a space in the rack into which a corresponding serverrack sub-assembly 218 can be placed and removed. For example, a serverrack sub-assembly 218 can be supported on rails that project fromopposite sides of the rack 214, and which can define the position of theslots. Also, although multiple server rack sub-assemblies 218 areillustrated as mounted within the rack 214, there might be only a singleserver rack sub-assembly.

The slots, and the server rack sub-assemblies 218, can be oriented withthe illustrated horizontal arrangement (with respect to gravity) asshown in FIG. 2A. Alternatively, the slots, and the server racksub-assemblies 218, can be oriented vertically (with respect togravity). Where the slots are oriented horizontally, they may be stackedvertically in the rack 214, and where the slots are oriented vertically,they may be stacked horizontally in the rack 214.

Server rack 214, as part of a larger data center for instance, mayprovide data processing and storage capacity. In operation, a datacenter may be connected to a network, and may receive and respond tovarious requests from the network to retrieve, process, and/or storedata. In operation, for example, the server rack 214 typicallyfacilitates the communication of information over a network with userinterfaces generated by web browser applications of users who requestservices provided by applications running on computers in thedatacenter. For example, the server rack 214 may provide or help providea user who is using a web browser to access web sites on the Internet orthe World Wide Web.

The server rack sub-assembly 218 may be one of a variety of structuresthat can be mounted in a server rack 214. For example, in someimplementations, the server rack sub-assembly 218 may be a “tray” ortray assembly that can be slidably inserted into the server rack 214.The term “tray” is not limited to any particular arrangement, butinstead applies to motherboard or other relatively flat structuresappurtenant to a motherboard for supporting the motherboard in positionin a rack structure. In some implementations, the server racksub-assembly 218 may be a server chassis, or server container (e.g.,server box). In some implementations, the server rack sub-assembly 218may be a hard drive cage.

Each server rack sub-assembly 218 can include a frame or cage, a printedcircuit board, e.g., motherboard, supported on the frame, and one ormore electronic devices 220, e.g., a processor or memory, mounted on theprinted circuit board. The electronic devices 220 can include, forinstance, processors, memories, hard drives, network switches, or otherIT components. Other appurtenances, such as cooling devices, fans,uninterruptible power supplies (UPS) (e.g., battery modules), powerconversion devices (e.g., from DC to DC, from DC to AC, from AC to DC)can be mounted to the server rack sub-assembly 218 (or otherwise to arack 214).

With respect specifically to FIG. 2A, the cooling units 216 arepositioned adjacent the server racks 214 in the bays 212. In someaspects, each cooling unit 216 may comprise a form factor (e.g.,dimensions such as width and height) that is similar to or the same as aform factor (e.g., dimensions such as width and height) of the serverracks 214. In some aspects, while a width and a height of the coolingunits 216 are the same as or similar to the width and height,respectively, of the server racks 214, a depth of the cooling unit(e.g., as shown in FIG. 2B), may be different (e.g., larger) than adepth of the server racks 214. Thus, in some aspects, each cooling unit216 may be positioned in any of the bays 212, and each server rack 214may be positioned in any of the bays 212, thereby making the coolingunits 216 and server racks 214 interchangeable within the frame assembly210.

In some aspects, the cooling units 216 may be fluidly coupled to asource of the cooling liquid, such as a chiller plant, one or moreevaporative cooling units (e.g., cooling towers), one or more condensingunits (e.g., in the case of direct expansion cooling), a natural sourceof cooling liquid (e.g., lake, ocean, river, or other natural body ofwater), or a combination thereof. In some aspects, the cooling units 216may be stand-alone refrigerant-based (DX) cooling units fluidly coupledto one or more condensing units located external to the data centerbuilding 202 (e.g., conventionally known as “CRAC” units).

As shown in more detail in FIG. 2B, a warm air aisle 218 is definedbetween adjacent rows 208 of server rack frame assemblies 210, with openback sides 224 of the server rack frame assemblies 210 facing the warmair aisle 218. Although only two rows 208 and one warm air aisle areshown in FIG. 2B, the data center building 202 may house multiple rows208 with warm air aisles 218 defined between pairs of rows 208 of theserver rack frame assemblies 210. In this example implementation, theopen back sides 224 of the frame assemblies 210 allow for airflow withminimal or no pressure gradient between the backs of the server racks214 (which are also open) that face the warm air aisle 218 and the warmair aisle 218 itself. Similarly, the server rack frame assemblies 210have, in this example, open front sides 222 that face the cool airaisles 220. The open front sides 222 of the frame assemblies 210 allowfor airflow with minimal or no pressure gradient between the fronts ofthe server racks 214 (which are also open) that face the cool air aisle220 and the cool air aisle 220 itself. Thus, in some aspects, an airpressure at the fronts and backs of the server racks 214, which are opento the aisles 220 and 218, respectively, are substantially equal orequal to an air pressure within the aisles 220 and 218, respectively.

In operation, the cooling units 216 circulate a cooling airflow 228through the front sides 222 of the server racks 214 (e.g., that are opento the human-occupiable workspace 204. The cooling airflow 228 receivesheat from electronic devices 220 in the racks 214 and warms the airflow228 to a heated airflow 226 that enters the warm air aisle 218. Theheated airflow 226 is drawn back into the cooling units 216 (e.g., byfans in the units 216) and cooled through the one or more cooling coils(e.g., by a flow of the chilled liquid, condenser water, refrigerant, oran electrically-powered cooler such as a Peltier cooler). The cooledairflow 228 is circulated (e.g., by the fans) back into thehuman-occupiable workspace 204 adjacent the front sides 222 of theserver racks 214 and server rack frame assemblies 210.

FIGS. 3A-3B are schematic illustrations of a side and end view,respectively, of another example implementation of a data center system300 that includes multiple, stacked and mixed computing and coolinglayers 313 a-313 c (collectively, an IT layer 344), positioned below anetworking layer 340, positioned in a human-occupiable workspace 304 ofa data center building 302. As shown in this example implementation, thedata center system 300 includes three layers 313 a-313 c of server racks314 (or server racks 315 or server racks 317) and cooling units 316, aswell as networking switches 330 positioned in the networking layer 340.The server racks 314 (and 315 and 317) and network switches 330 arepositioned in a frame assembly 310. In this example, the layers 313a-313 c and networking layer 340 are arranged in a row 308 (e.g., linearor substantially linear arrangement of the frame assembly 310). Inalternative arrangements, the layers 313 a-313 c and networking layer340 may be arranged in a non-linear frame assembly, such as, forinstance, a circular or partially circular frame assembly or otherwise.Although three layers 313 a-313 c are shown in this example, fewer(e.g., two) or more layers may be implemented without departing from thescope of the present disclosure. Regardless of the number of serverrack, or IT equipment, layers 313 a-313 c, in some aspects, a singlenetworking layer 340 may be positioned in the frame assembly 310 abovethe IT layers 344.

As shown in FIGS. 3A-3B, the networking layer 340 is positioned on topof the IT computing layers 313 a-313 c (and specifically, 313 c). Inthis example implementation, the networking layer 340 may be positionedon or include a support surface 346 that is positioned between the topcomputing layer 313 c and the networking layer 340. The support surface346, in some aspects, may be a raised support surface, such as amezzanine (e.g., with grated walking surfaces) from which a humanoperator may traverse and, e.g., service the networking switches 330 orthe illustrated networking cabling 332. In some aspects, the mezzaninemay be further supported by a data center floor 306 (e.g., slab, raisedfloor, or other surface) and/or one or more walls of the data centerbuilding 302. Thus, in some aspects, the support surface 346 may be ahuman-occupiable and traversable surface that is supported by, but notconsidered a part of, the data center building 302.

In alternative aspects, the support surface 356 may be part of orcomprise a floor of the data center building 302 (e.g., a second floor,third floor, or otherwise). For example, the support surface 346 may beor include a cement slab that can support, e.g., the networking layer340, and one or more humans or machines that may service the networkingswitches 330.

In this example, the networking layer 340 include network switches 330(e.g., stage 2 switches) positioned within bays 312 of the frameassembly 310 and connected, via network cabling 323 (e.g., fiber opticor otherwise) to the server racks 314 (and 315 and 317) in IT layers 344(e.g., one or more devices in the server racks). For example, in someaspects, particular network switches 330 in a particular bay 312 oflayer 340 may be coupled via the network cabling 332 to server racks 314(or 315 or 317) that are located in a particular bay 312 of IT layer 344that is directly below (e.g., vertically) the particular bay 312 oflayer 340. In some aspects, particular network switches 330 in aparticular bay 312 of layer 340 may be coupled via the network cabling332 to server racks 314 (or 315 or 317) that are located in a particularbay 312 of IT layer 344 that is indirectly below (e.g., vertically belowand horizontally offset by one bay 312) the particular bay 312 of ITlayer 340. Even further, in some aspects, particular network switches330 in a particular bay 312 of IT layer 344 may be coupled via thenetwork cabling 332 to server racks 314 (or 315 or 317) that are locatedin a particular bay 312 of IT layer 344 that is indirectly below (e.g.,vertically below and horizontally offset by more than one bay 312) theparticular bay 312 of layer 340. In short, networking switches 330 maybe connected via the cabling 332 to server racks 314 (and 315 and 317)that are vertically below, as well as vertically below and offset.

As shown in FIGS. 3A-3B, the networking cabling 332, and thus thecommunication connections between server racks 314, 315, and 317 (e.g.,through the network switches 330), between server racks 314 (and 315 and317) and a network external to the data center 300 (e.g., throughnetwork switches 330), and between network switches 330 (e.g., throughserver racks 314, 315, 317, or from/to an external network), are largelyunimpeded by, e.g., cooling infrastructure, such as cooling units 316,ductwork, piping, electrical cabling, or other cooling equipment, due tothe positioning of the network switches 330 in the layer 340. Bymounting the networking layer 340 vertically above (or, possibly,vertically below in a sub-floor, basement, or other level below the datacenter floor 306) the IT equipment layers 313 a-313 c, the Z, orvertical, dimension of the human-occupiable workspace 304 may be takenadvantage of as opposed to conventional data centers. This may alsoprovide for decreased lengths of the network cabling 332 (e.g., fiberoptic or otherwise) between switches 330 and server racks 314 (and 315and 317), switches 330 and a connection at the data center 300 to anexternal network, and between server racks 314 (and 315 and 317).

In some aspects, the example implementation of the data center 300 withseparated (e.g., vertically) IT equipment layer 344 from networkinglayer 340 may also increase a deployment velocity of the data centerequipment (e.g., server racks 314, 315, and 317, networking switches330) by separating networking routing paths (e.g., for the cabling 332)to enable easier incremental network turn ups, decommissions, andtransformations. This may also improve accessibility for the networkcabling 332. Such decoupling of the IT layer 344 and network layer 340may also allow separate construction trades (e.g., electrical andcabling installation) to occur in parallel.

The frame assembly 310 comprises structural members (e.g., metal ornon-metal, such as composite) arranged and connected to form multiplebays 312 in the frame assembly 310. Each bay 312, in this exampleimplementation, defines a volume within the frame assembly 310 and maybe the same or similar in volume and perimeter dimensions (e.g., height“H” of 10 feet, width “W” of 10 feet, and depth “D” of 3 feet) as theother bays 312 in the layer 313 of the frame assembly 310. In someaspects, the perimeter dimensions of the bays 312 (and thus the volumesdefined by the bays 312) in the frame assembly 310 are based on orassociated with dimensions of the server racks 314 (or 315 or 317). Forexample, in the illustrated implementations, each bay 312 is sized(e.g., at least 10′ W and 10′ H) according to dimensions of a doubleserver rack 314 (or 315 or 317) (i.e., two server racks of 42U inheight, two server racks of between 18-22U in height, or other sizedserver racks). In some aspects, the server racks 314 (or 315 or 317) maybe 54U in height, although frame assembly 110 may handle a variation ofrack sizes. For example, the racks 314 (or 315 or 317) may be about 50inches wide each.

Furthermore, different combinations of racks 314 (or 315 or 317) can beused in the example implementation. For an example, four racks 314 (or315 or 317), each 24 inches in width, can be used within the frameassembly 310. The racks 314 (or 315 or 317) can also vary in depth. Forexample, the front faces of the server racks 314 (or 315 or 317) may beflush with a front sides of the assembly 310, while a rear of the racks314 (or 315 or 317) may extend various depths beyond back sides of theassembly 310.

As shown in FIG. 3A, groups of bays 312 may be arranged to form columns309 (e.g., a vertical group of three stacked bays 312). The bays 312 maybe further grouped, in this example, in clusters 311 a-311 d of bays 312that include a multi-dimensional (e.g., vertical and horizontal)grouping of bays 312. In this example implementation, clusters 311 a-311d are shown as three by three (e.g., three columns 309 of bays 312 bythree layers 313 of bays 312) groupings of nine total bays 312. In someaspects, a column 309 or cluster 311 of bays 312 may be part of a power,cooling liquid, or network sharing architecture. For example, in someaspects, separate electrical power conductors may provide electricalpower to the server racks 314 (or 315 or 317) and cooling units 316 thatare positioned in separate columns 309 or clusters 311 of bays 312,e.g., to ensure that a failure of one power conductor only affects theracks 314 (or 315 or 317) and/or cooling units 316 in that particularcolumn 309 or cluster 311. Likewise, in some aspects, separate coolingliquid conduits may provide a cooling liquid to the cooling units 316that are positioned in separate columns 309 or clusters 311 of bays 312,e.g., to ensure that a failure of one cooling liquid conduit onlyaffects the cooling units 316 in that particular column 309 or cluster311. Further, in some aspects, separate networking connectors mayprovide data communication to the server racks 314 (or 315 or 317) thatare positioned in separate columns 309 or clusters 311 of bays 312,e.g., to ensure that a failure of one network connector only affects theracks 314 (or 315 or 317) in that particular column 309 or cluster 311.

As shown in FIG. 3A, the server racks 314 (or 315 or 317) supportelectronic devices, such as processors, memory modules, networkingswitches, battery modules, and other server rack computing components,both physically by providing structure for the devices to be placed inand electrically by providing electric power to the devices from a mainsource of power (e.g., through an inverter, a transformer, or both).Generally, each illustrated server rack 314 (or 315 or 317) may be oneof a number of server racks within the data center building 302, whichmay include a server farm or a co-location facility that containsvarious rack mounted computer systems. Each server rack 314 (or 315 or317) may define multiple slots that are arranged in an orderly andrepeating fashion within the server rack 314 (or 315 or 317), and eachslot is a space in the rack into which a corresponding server racksub-assembly 318 can be placed and removed. For example, a server racksub-assembly 318 can be supported on rails that project from oppositesides of the rack 314 (or 315 or 317), and which can define the positionof the slots. Also, although multiple server rack sub-assemblies 318 areillustrated as mounted within the rack 314 (or 315 or 317), there mightbe only a single server rack sub-assembly.

The slots, and the server rack sub-assemblies 314, can be oriented withthe illustrated horizontal arrangement (with respect to gravity) asshown in FIG. 3A. Alternatively, the slots, and the server racksub-assemblies 318, can be oriented vertically (with respect togravity). Where the slots are oriented horizontally, they may be stackedvertically in the rack 314 (or 315 or 317), and where the slots areoriented vertically, they may be stacked horizontally in the rack 314(or 315 or 317).

Server rack 314 (or 315 or 317), as part of a larger data center forinstance, may provide data processing and storage capacity. Inoperation, a data center may be connected to a network, and may receiveand respond to various requests from the network to retrieve, process,and/or store data. In operation, for example, the server rack 314 (or315 or 317) typically facilitates the communication of information overa network with user interfaces generated by web browser applications ofusers who request services provided by applications running on computersin the datacenter. For example, the server rack 314 (or 315 or 317) mayprovide or help provide a user who is using a web browser to access websites on the Internet or the World Wide Web.

The server rack sub-assembly 318 may be one of a variety of structuresthat can be mounted in a server rack 314 (or 315 or 317). For example,in some implementations, the server rack sub-assembly 318 may be a“tray” or tray assembly that can be slidably inserted into the serverrack 314 (or 315 or 317). The term “tray” is not limited to anyparticular arrangement, but instead applies to motherboard or otherrelatively flat structures appurtenant to a motherboard for supportingthe motherboard in position in a rack structure. In someimplementations, the server rack sub-assembly 318 may be a serverchassis, or server container (e.g., server box). In someimplementations, the server rack sub-assembly 318 may be a hard drivecage.

Each server rack sub-assembly 318 can include a frame or cage, a printedcircuit board, e.g., motherboard, supported on the frame, and one ormore electronic devices 320, e.g., a processor or memory, mounted on theprinted circuit board. The electronic devices 320 can include, forinstance, processors, memories, hard drives, network switches, or otherIT components. Other appurtenances, such as cooling devices, fans,uninterruptible power supplies (UPS) (e.g., battery modules), powerconversion devices (e.g., from DC to DC, from DC to AC, from AC to DC)can be mounted to the server rack sub-assembly 318 (or otherwise to arack 314 (or 315 or 317)).

Server racks 314, server racks 315, and server racks 317, as shown, maybe physically similar or identical (e.g., similar or identical height,width, depth, weight), but may provide different computing power and,thus, different heat output. For example, server racks 314 may be 15 kWracks, with electronic devices 320 that generate about 15 kW of heateach. Server racks 315 may be 30 kW racks, with electronic devices 320that generate about 30 kW of heat each. Server racks 317 may be 100 kWracks, with electronic devices 320 that generate about 100 kW of heateach. Thus, in some aspects, a number and computing power of theparticular racks (e.g., server racks 314, 315, and/or 317) within aparticular layer 313 a-313 c, a particular column 309, or a particularcluster 311 a-311 d of bays 312 may dictate a number of cooling units316 that are positioned within the particular layer 313 a-313 c,particular column 309, or particular cluster 311 a-311 d of bays 312.For example, the greater number of server racks 317 relative to serverracks 314 within any particular layer, column, or cluster may dictate(e.g., due to cooling requirements for the heat generated by theelectronic devices 320 in the racks) a greater number of cooling units316 within the particular layer, column, or cluster. Conversely, thegreater number of server racks 314 relative to server racks 317 withinany particular layer, column, or cluster may dictate (e.g., due tocooling requirements for the heat generated by the electronic devices320 in the racks) a fewer number of cooling units 316 within theparticular layer, column, or cluster.

In some aspects, each particular layer, column, or cluster (or othersub-unit of the frame assembly 310) may be reconfigured over anoperating lifetime of the data center system 300. For example,reconfiguration may include moving particular server racks 314 (or 315or 317) from particular bays 312 to other bays 312. In suchreconfigurations, a number of cooling units 316 may not change, but suchcooling units 312 may also be moved from particular bays 312 to otherbays 312 (e.g., to account for movement of heat sources from onelocation to another location). Reconfigurations may also includereplacing server racks 314 with higher power server racks 315 or 317within particular bays 312. In such reconfigurations, a number ofcooling units 316 may increase (e.g., to account for additional heatgenerated by the higher power racks 315 or 317) within a particularlayer, column, or cluster. Reconfigurations may also include replacingserver racks 317 with lower power server racks 315 or 314 withinparticular bays 312. In such reconfigurations, a number of cooling units316 may decrease (e.g., to account for less heat generated by the lowerpower racks 315 or 314) within a particular layer, column, or cluster.Reconfigurations may also include removing server racks 314 (or 315 or317) from particular bays 312. In such reconfigurations, a number ofcooling units 316 may decrease (e.g., to account for less heatgenerated) within a particular layer, column, or cluster. In suchreconfigurations, for example, when a particular bay 312 does notinclude any server rack 314 (or 315 or 317) or cooling unit 316, ablank-off panel (e.g., sheet metal or otherwise) may be installed acrossa width and height of the bay 312 to prevent airflow from beingcirculated through the empty bay 312.

While FIG. 1B shows the clusters 311 a-311 d within a single row 308 ofthe frame assembly 310, each cluster 311 may represent a particularconfiguration at a particular operating time of the data center system300 for illustrative purposes. For example, cluster 311 a may representa configuration at an initial operating time of the system 300, withonly four server racks 314 and one cooling unit 316 within the ninetotal bays 312 of the cluster 311 a. At the initial operating time, onlya single cooling unit 316 may be needed to remove the heat generated bythe four server racks 314. Empty bays 312 may include blank-off panelsas previously described.

Cluster 311 b may represent a configuration at a later operating time ofthe system 300, with eight server racks 314 and one cooling unit 316within the nine total bays 312 of the cluster 311 b. At the lateroperating time, still only a single cooling unit 316 may be needed toremove the heat generated by the eight server racks 314. In thisconfiguration, no bays 312 are empty.

Cluster 311 c may represent a configuration at another later operatingtime of the system 300, with seven server racks 315 and two coolingunits 316 within the nine total bays 312 of the cluster 311 c. At thislater operating time, two single cooling units 316 may be needed toremove the heat generated by the seven server racks 315, which, asdescribed, may have more computing power and thus generate more heatthan the server racks 314, and thus require additional cooling power(through the additional cooling units 316) to remove the generated heat.

Cluster 311 d may represent a configuration at still another lateroperating time of the system 300, with six server racks 317 and threecooling units 316 within the nine total bays 312 of the cluster 311 d.At this later operating time, three single cooling units 316 may beneeded to remove the heat generated by the six server racks 317, which,as described, may have more computing power and thus generate more heatthan the server racks 314 and 317, and thus require additional coolingpower (through the additional cooling units 316) to remove the generatedheat.

With respect specifically to FIG. 3A, the cooling units 316 arepositioned adjacent the server racks 314 (or 315 or 317) in the bays312. In some aspects, each cooling unit 316 may comprise a form factor(e.g., dimensions such as width and height) that is similar to or thesame as a form factor (e.g., dimensions such as width and height) of theserver racks 314 (or 315 or 317). In some aspects, while a width and aheight of the cooling units 316 are the same as or similar to the widthand height, respectively, of the server racks 314 (or 315 or 317), adepth of the cooling unit (e.g., as shown in FIG. 3B), may be different(e.g., larger) than a depth of the server racks 314 (or 315 or 317).Thus, in some aspects, each cooling unit 316 may be positioned in any ofthe bays 312, and each server rack 314 (or 315 or 317) may be positionedin any of the bays 312, thereby making the cooling units 316 and serverracks 314 (or 315 or 317) interchangeable within the frame assembly 310.

In some aspects, the cooling units 316 may be fluidly coupled to asource of the cooling liquid, such as a chiller plant, one or moreevaporative cooling units (e.g., cooling towers), one or more condensingunits (e.g., in the case of direct expansion cooling), a natural sourceof cooling liquid (e.g., lake, ocean, river, or other natural body ofwater), or a combination thereof. In some aspects, the cooling units 316may be stand-alone refrigerant-based (DX) cooling units fluidly coupledto one or more condensing units located external to the data centerbuilding 302 (e.g., conventionally known as “CRAC” units.

As shown in more detail in FIG. 3B, a warm air aisle 318 is definedbetween adjacent rows 308 of server rack frame assemblies 310, with openback sides 324 of the server rack frame assemblies 310 facing the warmair aisle 318. Although only two rows 308 and one warm air aisle areshown in FIG. 3B, the data center building 302 may house multiple rows308 with warm air aisles 318 defined between pairs of rows 308 of theserver rack frame assemblies 310. In this example implementation, theopen back sides 324 of the frame assemblies 310 allow for airflow withminimal or no pressure gradient between the backs of the server racks314 (or 315 or 317) (which are also open) that face the warm air aisle318 and the warm air aisle 318 itself. Similarly, the server rack frameassemblies 310 have, in this example, open front sides 322 that face thecool air aisles 320. The open front sides 322 of the frame assemblies310 allow for airflow with minimal or no pressure gradient between thefronts of the server racks 314 (or 315 or 317) (which are also open)that face the cool air aisle 320 and the cool air aisle 320 itself.Thus, in some aspects, an air pressure at the fronts and backs of theserver racks 314 (or 315 or 317), which are open to the aisles 320 and318, respectively, are substantially equal or equal to an air pressurewithin the aisles 320 and 318, respectively.

In operation, the cooling units 316 circulate a cooling airflow 328through the front sides 322 of the server racks 314 (or 315 or 317)(e.g., that are open to the human-occupiable workspace 304. The coolingairflow 328 receives heat from electronic devices 320 in the racks 314(or 315 or 317) and warms the airflow 328 to a heated airflow 326 thatenters the warm air aisle 318. The heated airflow 326 is drawn back intothe cooling units 316 (e.g., by fans in the units 316) and cooledthrough the one or more cooling coils (e.g., by a flow of the chilledliquid, condenser water, refrigerant, or an electrically-powered coolersuch as a Peltier cooler). The cooled airflow 328 is circulated (e.g.,by the fans) back into the human-occupiable workspace 304 adjacent thefront sides 322 of the server racks 314 (or 315 or 317) and server rackframe assemblies 310.

FIGS. 4A-4B are schematic illustrations of a side and back view,respectively, of an example implementation of a data center system,e.g., for data center 300, that includes one or more automatic servicingmachines, such as the illustrated automated service machines 400 and450. Generally, each of the automated service machines 400 and 450 maybe operated, controlled, programmed, or pre-programmed to perform one ormore tasks related to deployment, servicing, or replacement of ITequipment, such as server rack-sub-assemblies 318 or electronic devices320 or networking equipment, such as networking switches 330. Althoughthe automated service machines 400 and 450 are illustrated in thecontext of the data center 300, such machines 400 and 450 according tothe present disclosure may also be utilized, e.g., in data center 100,data center 200, or another data center according to the presentdisclosure (e.g., a data center with a layered, stacked structure of ITequipment and networking equipment).

The illustrated example includes an automated service machine 400 and anautomated service machine 450 within the data center 300. Generally,automated service machine 400 represents a human-operated (e.g., humanon-board) automated service machine (e.g., a lift truck, forklift, orotherwise), and includes a motor 415 (e.g., electric, propane, gasoline,or otherwise) mounted on rollers 420 (e.g., wheels, sliders, railwheels). The automated service machine 400 also includes a supportbasket 410 in which a human-operator may ride while, for example,operating the automated service machine 400. A mast 405 is mounted tothe motor 415 and provides a height-adjustment mechanism for the basket410 (which may vertically move up and down the mast 405 to adjust theheight of the basket 410 relative to the data center floor 306).Although the rollers 420 are shown as contacting the data center floor306, in alternative implementations, the rollers 420 may engage one ormore tracks (or slots) attached to (or formed in) the data center floor306. In some aspects, the tracks (not shown here) may be arranged to runalong one or both sides (front and/or back sides) of one or more rows ofthe layers 344 and 340 in the data center 300.

Generally, automated service machine 450 represents a human orcomputer-controlled (e.g., no human on-board) automated service machine,and includes a motor 465 (e.g., electric, propane, gasoline, orotherwise) mounted on rollers 470 (e.g., wheels, sliders, rail wheels).The automated service machine 450 also includes a mast 455 that ismounted to the motor 465 and provides a height-adjustment mechanism foran extendable end effect 460 (e.g., claw, pincher, forks, magnetizedend, etc.), which may vertically move up and down the mast 455 to adjustthe height of the end effect 460 relative to the data center floor 306.Although the rollers 470 are shown as contacting the data center floor306, in alternative implementations, the rollers 470 may engage one ormore tracks (or slots) attached to (or formed in) the data center floor306. In some aspects, the tracks (not shown here) may be arranged to runalong one or both sides (front and/or back sides) of one or more rows ofthe layers 344 and 340 in the data center 300.

In some aspects, one or both of the automated service machines 400 and450 may at least partially be controlled by a control system 402,through one or more wireless (or wired) signals 404. The control system402, in some aspects, may be a micro-processor based control system,such as a building automation system or otherwise. In any event, whethercontrolled solely by a human operator on board, such as for automatedservice machine 400, or with or by control system 402 (which may be onboard the automated service machine 400 or 450), such as for automatedservice machine 450, each of the automated service machines 400 and 450may perform one or more of the following operations.

For example, the automated service machines 400 and 450 may be movedthrough the data center 300 adjacent any of the bays 312 of the frameassembly 310, whether in IT layers 344 or networking layer 340. Themovement of the automated service machines 400 and 450 can be betweenrows of the frame assemblies 310 (and layers 344 and 340) or otherwisewithin a human-occupiable workspace of the data center 300. In someaspects, the automated service machines 400 and 450 may be moved throughthe data center 300 adjacent any of the bays 312 of the frame assembly310 based on an indication by one or more of the serverrack-sub-assemblies 318, electronic devices 320 or networking switches330, of a malfunction or need for replacement. In some aspects, theautomated service machines 400 and 450 may be moved through the datacenter 300 adjacent any of the bays 312 of the frame assembly 310 basedon a pre-determined periodic maintenance of one or more of the serverrack-sub-assemblies 318, electronic devices 320 or networking switches330.

Once the automated service machines 400 and 450 are moved into aparticular place in the data center 300 (e.g., adjacent a particular bay312 of the frame assembly 310), the automated service machines 400 and450 may be controlled or operated to perform a deployment, servicing, orreplacement function, among others. For example, if a particularcomponent such as a server rack-sub-assembly 318, electronic device 320,or networking switch 330 needs to be deployed, serviced, or replaced, alocation within the data center 300 of the component (three dimensionallocation, including height from the data center floor 306) may bedetermined (e.g., by the control system 402, a human-operator on orcontrolling one of the automated service machines 400 or 450, orotherwise). For example, the control system 402 may determine aparticular location (e.g., in three-dimensional space) of each componentin the human-occupiable workspace 304 of the data center building 302(e.g., with a GPS receiver positioned at or near each of the server racksub-assemblies 318 or server racks 314, a pre-determined mapping of eachof the server rack sub-assemblies 318 or server racks 314 to theirassociated locations, or otherwise).

If, for instance, the determined three-dimensional location of theparticular component (e.g., to be deployed, serviced, replaced, orotherwise) is above (e.g., in a Z, or height, direction) than isreachable by a human standing on the data center floor 306, then theautomated service machines 400 and 450 may be operated or moved to raisethe particular component to the appropriate height. For example, thesupport basket 410 of automated service machine 400 may be adjusted(e.g., vertically on the mast 405) to raise a human-operator within thebasket 410 to a particular height above the data center floor 306 to,e.g., deploy, service, or replace the particular component. The endeffect 460 (which may hold or otherwise be connected to the particularcomponent) of automated service machine 450 may be adjusted (e.g.,vertically on the mast 455 and/or horizontally from the mast 455) toraise the particular component supported or held by the end effect 460to a particular height above the data center floor 306 to, e.g., deploy,service, or replace the particular component.

Other operations may be performed by the automated services machines 400and 450. For example, the automated service machines 400 and 450 canalso be used to deploy non-IT equipment as well, such as air baffles orshelving that may be needed in an area or during a reconfiguration. Themachines 400 and 450 may also deploy power conversion and batteries aswell. For example, the automated machines 400 and 450 can also plug inand power up a rack, as well as move and reconfigure a rack (e.g., aserver rack).

FIG. 5 is a schematic illustration of an example controller 500 (orcontrol system) for an automated service machine according to thepresent disclosure. For example, the controller 500 may include or bepart of the control system 402 shown in FIGS. 4A-4B. The controller 500is intended to include various forms of digital computers, such asprinted circuit boards (PCB), processors, digital circuitry, orotherwise that is part of a vehicle. Additionally the system can includeportable storage media, such as, Universal Serial Bus (USB) flashdrives. For example, the USB flash drives may store operating systemsand other applications. The USB flash drives can include input/outputcomponents, such as a wireless transmitter or USB connector that may beinserted into a USB port of another computing device.

The controller 500 includes a processor 510, a memory 520, a storagedevice 530, and an input/output device 540. Each of the components 510,520, 530, and 540 are interconnected using a system bus 550. Theprocessor 510 is capable of processing instructions for execution withinthe controller 500. The processor may be designed using any of a numberof architectures. For example, the processor 510 may be a CISC (ComplexInstruction Set Computers) processor, a RISC (Reduced Instruction SetComputer) processor, or a MISC (Minimal Instruction Set Computer)processor.

In one implementation, the processor 510 is a single-threaded processor.In another implementation, the processor 510 is a multi-threadedprocessor. The processor 510 is capable of processing instructionsstored in the memory 520 or on the storage device 530 to displaygraphical information for a user interface on the input/output device540.

The memory 520 stores information within the controller 500. In oneimplementation, the memory 520 is a computer-readable medium. In oneimplementation, the memory 520 is a volatile memory unit. In anotherimplementation, the memory 520 is a non-volatile memory unit.

The storage device 530 is capable of providing mass storage for thecontroller 500. In one implementation, the storage device 530 is acomputer-readable medium. In various different implementations, thestorage device 530 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 540 provides input/output operations for thecontroller 500. In one implementation, the input/output device 540includes a keyboard and/or pointing device. In another implementation,the input/output device 540 includes a display unit for displayinggraphical user interfaces.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The apparatus can be implemented in a computerprogram product tangibly embodied in an information carrier, forexample, in a machine-readable storage device for execution by aprogrammable processor; and method steps can be performed by aprogrammable processor executing a program of instructions to performfunctions of the described implementations by operating on input dataand generating output. The described features can be implementedadvantageously in one or more computer programs that are executable on aprogrammable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. A computer program is a set of instructionsthat can be used, directly or indirectly, in a computer to perform acertain activity or bring about a certain result. A computer program canbe written in any form of programming language, including compiled orinterpreted languages, and it can be deployed in any form, including asa stand-alone program or as a module, component, subroutine, or otherunit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor will receive instructionsand data from a read-only memory or a random access memory or both. Theessential elements of a computer are a processor for executinginstructions and one or more memories for storing instructions and data.Generally, a computer will also include, or be operatively coupled tocommunicate with, one or more mass storage devices for storing datafiles; such devices include magnetic disks, such as internal hard disksand removable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube)or LCD (liquid crystal display) monitor for displaying information tothe user and a keyboard and a pointing device such as a mouse or atrackball by which the user can provide input to the computer.Additionally, such activities can be implemented via touchscreenflat-panel displays and other appropriate mechanisms.

The features can be implemented in a control system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include a local area network (“LAN”),a wide area network (“WAN”), peer-to-peer networks (having ad-hoc orstatic members), grid computing infrastructures, and the Internet.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features that are described in this specification inthe context of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, exampleoperations, methods, or processes described herein may include moresteps or fewer steps than those described. Further, the steps in suchexample operations, methods, or processes may be performed in differentsuccessions than that described or illustrated in the figures.Accordingly, other implementations are within the scope of the followingclaims.

1. A data center system, comprising: a frame assembly that comprises aplurality of bays defined along a lengthwise dimension of the frameassembly, the plurality of bays arranged in a plurality of stackedlayers of bays, the plurality of stacked layers comprising at least afirst layer of bays and a second layer of bays positioned verticallyabove the first layer of bays; a plurality of server racks positioned inthe bays of the first layer of bays, each of the server racks configuredto support a plurality of data center server devices that define aparticular amount of computing power; and a plurality of networkswitches positioned in the bays of the second layer of bays and externalof the plurality of server racks positioned in the first layer of bays,each of the network switches communicably coupled to at least one of thedata center server devices in the first layer of bays, each of theplurality of network switches comprising a stage 2 aggregation switch.2. The data center system of claim 1, wherein the plurality of bays arefurther arranged in a third layer of bays that is positioned verticallybetween the first and the second layers of bays; and another pluralityof server racks are positioned in the bays of the third layer of bays,each of the other plurality server racks is configured to supportanother plurality of data center server devices that define anadditional amount of computing power.
 3. The data center system of claim1, further comprising at least one cooling unit sized for a bay of theplurality of bays of the frame assembly and configured to circulate aheated airflow from an open back side of a particular server rack, coolthe heated air, and circulate a cooling airflow through an open frontside of the particular server rack.
 4. The data center system of claim1, wherein each bay comprises a volume defined at least in part by aspecified height that is orthogonal to the lengthwise dimension and aspecified width that is parallel to the lengthwise dimension and sizedto at least partially enclose at least one server rack.
 5. The datacenter system of claim 4, wherein the specified width and the specifiedheight of the bay are based on a form factor of the at least one serverrack.
 6. The data center system of claim 3, wherein at least oneperimeter dimension of the cooling unit is based on a form factor of theat least one server rack.
 7. The data center system of claim 1, whereinthe at least one server rack comprises at least two server racks, eachof the at least two server racks providing between 5 kW and 100 kW ofcomputing power.
 8. The data center system of claim 1, wherein at leasta portion of the frame assembly that defines the bays of the secondlayer of bays is configured to mount on a support surface of a datacenter building.
 9. The data center system of claim 8, wherein thesupport surface comprises a structural steel mezzanine.
 10. The datacenter system of claim 2, wherein the frame assembly comprises: a firstportion that defines the first layer of bays; a second portion stackedon top of the first portion that defines the second layer of bays; and athird portion stacked on top of the second portion that defines thethird layer of bays.
 11. A method of managing a data center comprising:operating a plurality of data center server devices that define aparticular amount of computing power, the plurality of data centerserver devices supported in a plurality of server racks positioned in aframe assembly that comprises a plurality of bays defined along alengthwise dimension of the frame assembly, the plurality of baysarranged in a plurality of stacked layers of bays, the plurality ofstacked layers comprising at least a first layer of bays and a secondlayer of bays positioned vertically above the first layer of bays, theplurality of server racks positioned in the bays of the first layer ofbays; and operating a plurality of network switches that are positionedin the bays of the second layer of bays and external of the plurality ofserver racks positioned in the first layer of bays, each of the networkswitches communicably coupled to at least one of the data center serverdevices in the first layer of bays, and each of the plurality of networkswitches comprises a stage 2 aggregation switch.
 12. The method of claim11, wherein the plurality of bays are further arranged in a third layerof bays that is positioned vertically between the first and the secondlayers of bays, the method further comprising: operating an additionalplurality of data center server devices that define an additional amountof computing power, the additional plurality of data center serverdevices mounted in an additional plurality of server racks positioned inthe bays of the third layer of bays.
 13. The method of claim 11, furthercomprising operating at least one cooling unit sized for a bay of theplurality of bays of the frame assembly to circulate a heated airflowfrom an open back side of a particular server rack, cool the heated air,and circulate a cooling airflow through an open front side of theparticular server rack.
 14. The method of claim 11, wherein each baycomprises a volume defined at least in part by a specified height thatis orthogonal to the lengthwise dimension and a specified width that isparallel to the lengthwise dimension and sized to at least partiallyenclose at least one server rack, and the specified width and thespecified height of the bay are based on a form factor of at least oneserver rack.
 15. The method of claim 11, wherein at least one of theplurality of server racks comprises at least two server racks, each ofthe two server racks providing between 5 kW and 100 kW of computingpower.
 16. The method of claim 12, further comprising: deploying theframe assembly into a human-occupiable space of a data center building;arranging a first portion of the frame assembly that defines the firstlayer of bays onto a data center floor of the data center building;arranging a third portion of the frame assembly that defines the thirdlayer of bays on top of the first portion of the frame assembly;arranging a second portion of the frame assembly that defines the secondlayer of bays on top of the third portion of the frame assembly;deploying at least one cooling unit into at least one bay of theplurality of bays of the frame assembly; deploying at least theplurality and the additional plurality of server racks into the firstand third layers of bays defined by the first and third portions of theframe assembly; deploying the plurality of network switches into thesecond layer of bays defined by the second portion of the frameassembly; and communicably coupling the plurality of network switcheswith the plurality and the additional plurality of server racks.
 17. Themethod of claim 16, further comprising mounting at least a part of thesecond portion of the frame assembly to a support surface of the datacenter building.
 18. The method of claim 17, wherein the support surfacecomprises a structural steel mezzanine.
 19. The method of claim 16,further comprising: deploying, to at least one of the first or thirdlayers of bays in the first and third portions of the frame assembly, anincremental amount of data center server devices supported in anincremental number of server racks; based on the deploying of theincremental amount of data center server devices, deploying anincremental amount of network switches to the second layer of bays inthe second portion of the frame assembly; and communicably coupling theincremental amount of network switches to the incremental amount of datacenter server devices.
 20. The method of claim 19, further comprising,based on the deploying of the incremental amount of data center serverdevices, deploying at least one additional cooling unit into at leastone additional bay of the plurality of bays of the frame assembly.